High speed non-concurrency controlled database

ABSTRACT

Embodiments of the present invention provide a method and system for high-speed database searching with concurrent updating, without the use of database locks or access controls, for large database systems. Specifically, a plurality of search queries may be received over a network, the database may be searched, and a plurality of search replies may be sent over the network. While searching the database, new information received over the network may be incorporated into the database by creating a new element based on the new information and writing a pointer to the new element to the database using a single uninterruptible operation.

CLAIM FOR PRIORITY/CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation of U.S. application Ser. No. 10/285,618 filed 01Nov. 2002, which claims the benefit of U.S. provisional application No.60/330,842, filed Nov. 1, 2001, which is incorporated by reference inits entirety, and U.S. provisional application No. 60/365,169, filedMar. 19, 2002, which is incorporated by reference in its entirety. Thisapplication is related to U.S. non-provisional applications Ser. No.10/285,575 filed 01 Nov. 2002, now U.S. Pat. No. 6,681,228 B2, Ser. No.10/285,544 filed 01 Nov. 2002, Ser. No. 10/285,549 filed 01 Nov. 2002,and Ser. No. 10/285,547 filed 01 Nov. 2002.

TECHNICAL FIELD

This disclosure relates to computer systems. More specifically, thisdisclosure relates to a method and system for providing high-speeddatabase searching with concurrent updating, without the use of databaselocks or access controls, for large database systems.

BACKGROUND OF THE INVENTION

As the Internet continues its meteoric growth, scaling domain nameservice (DNS) resolution for root and generic top level domain (gTLD)servers at reasonable price points is becoming increasingly difficult.The A root server (i.e., a.root-server.net) maintains and distributesthe Internet namespace root zone file to the 12 secondary root serversgeographically distributed around the world (i.e., b.root-server.net,c.root-server.net, etc.), while the corresponding gTLD servers (i.e.,a.gtld-servers.net, b.gtld-servers.net, etc.) are similarly distributedand support the top level domains (e.g., *.com, *.net, *.org, etc.). Theever-increasing volume of data coupled with the unrelenting growth inquery rates is forcing a complete rethinking of the hardware andsoftware infrastructure needed for root and gTLD DNS service over thenext several years. The typical single server installation of thestandard “bind” software distribution is already insufficient for thedemands of the A root and will soon be unable to meet even gTLD needs.With the convergence of the public switched telephone network (PSTN) andthe Internet, there are opportunities for a general purpose, highperformance search mechanism to provide features normally associatedwith Service Control Points (SCPs) on the PSTN's SS7 signaling networkas new, advanced services are offered that span the PSTN and theInternet, including Advanced Intelligent Network (AIN), Voice OverInternet Protocol (VoIP) services, geolocation services, etc.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a system block diagram, according to an embodiment of thepresent invention.

FIG. 2 is a detailed block diagram that illustrates a message datastructure, according to an embodiment of the present invention.

FIG. 3 is a detailed block diagram that illustrates a message latencydata structure architecture, according to an embodiment of the presentinvention.

FIG. 4 is a detailed block diagram that illustrates a non-concurrencycontrolled data structure architecture, according to an embodiment ofthe present invention.

FIG. 5 is a detailed block diagram that illustrates a non-concurrencycontrolled data structure architecture, according to an embodiment ofthe present invention.

FIG. 6 is a detailed block diagram that illustrates a non-concurrencycontrolled data structure architecture, according to an embodiment ofthe present invention.

FIG. 7 is a detailed block diagram that illustrates a non-concurrencycontrolled data structure architecture, according to an embodiment ofthe present invention.

FIG. 8 is a detailed block diagram that illustrates a non-concurrencycontrolled data structure architecture, according to an embodiment ofthe present invention.

FIG. 9 is a top level flow diagram that illustrates a method forsearching and concurrently updating a database without the use ofdatabase locks or access controls, according to an embodiment of thepresent invention.

DETAILED DESCRIPTION

Embodiments of the present invention provide a method and system forhigh-speed database searching with concurrent updating, without the useof database locks or access controls, for large database systems.Specifically, a plurality of search queries may be received over anetwork, the database may be searched, and a plurality of search repliesmay be sent over the network. While searching the database, newinformation received over the network may be incorporated into thedatabase by creating a new element based on the new information and,without locking the database, writing a pointer to the new element tothe database using a single uninterruptible operation.

FIG. 1 is a block diagram that illustrates a system according to anembodiment of the present invention. Generally, system 100 may host alarge, memory-resident database, receive search requests and providesearch responses over a network. For example, system 100 may be asymmetric, multiprocessing (SMP) computer, such as, for example, an IBMRS/6000® M80 or S80 manufactured by International Business MachinesCorporation of Armonk, N.Y., a Sun Enterprise™ 10000 manufactured by SunMicrosystems, Inc. of Santa Clara, Calif., etc. System 100 may also be amulti-processor personal computer, such as, for example, a CompaqProLiant™ ML530 (including two Intel Pentium® III 866 MHz processors)manufactured by Hewlett-Packard Company of Palo Alto, Calif. System 100may also include a multiprocessing operating system, such as, forexample, IBM AIX® 4, Sun Solaris™ 8 Operating Environment, Red HatLinux® 6.2, etc. System 100 may receive periodic updates over network124, which may be concurrently incorporated into the database.Embodiments of the present invention may achieve very high databasesearch and update throughput by incorporating each update to thedatabase without the use of database locks or access controls.

In an embodiment, system 100 may include at least one processor 102-1coupled to bus 101. Processor 102-1 may include an internal memory cache(e.g., an L1 cache, not shown for clarity). A secondary memory cache103-1 (e.g., an L2 cache, L2/L3 caches, etc.) may reside betweenprocessor 102-1 and bus 101. In a preferred embodiment, system 100 mayinclude a plurality of processors 102-1 . . . 102-P coupled to bus 101.A plurality of secondary memory caches 103-1 . . . 103-P may also residebetween plurality of processors 102-1 . . . 102-P and bus 101 (e.g., alook-through architecture), or, alternatively, at least one secondarymemory cache 103-1 may be coupled to bus 101 (e.g., a look-asidearchitecture). System 100 may include memory 104, such as, for example,random access memory (RAM), etc., coupled to bus 101, for storinginformation and instructions to be executed by plurality of processors102-1 . . . 102-P.

Memory 104 may store a large database, for example, for translatingInternet domain names into Internet addresses, for translating names orphone numbers into network addresses, for providing and updatingsubscriber profile data, for providing and updating user presence data,etc. Advantageously, both the size of the database and the number oftranslations per second may be very large. For example, memory 104 mayinclude at least 64 GB of RAM and may host a 500M (i.e., 500×10⁶) recorddomain name database, a 500M record subscriber database, a 450M recordtelephone number portability database, etc.

On an exemplary 64-bit system architecture, such as, for example, asystem including at least one 64-bit big-endian processor 102-1 coupledto at least a 64-bit bus 101 and a 64-bit memory 104, an 8-byte pointervalue may be written to a memory address on an 8-byte boundary (i.e., amemory address divisible by eight, or, e.g., 8N) using a single,uninterruptible operation. Generally, the presence of secondary memorycache 103-1 may simply delay the 8-byte pointer write to memory 104. Forexample, in one embodiment, secondary memory cache 103-1 may be alook-through cache operating in write-through mode, so that a single,8-byte store instruction may move eight bytes of data from processor102-1 to memory 104, without interruption, and in as few as two systemclock cycles. In another embodiment, secondary memory cache 1031 may bea look-through cache operating in write-back mode, so that the 8-bytepointer may first be written to secondary memory cache 103-1, which maythen write the 8-byte pointer to memory 104 at a later time, such as,for example, when the cache line in which the 8-byte pointer is storedis written to memory 104 (i.e., e.g., when the particular cache line, orthe entire secondary memory cache, is “flushed”).

Ultimately, from the perspective of processor 102-1, once the data arelatched onto the output pins of processor 102-1, all eight bytes of dataare written to memory 104 in one contiguous, uninterrupted transfer,which may be delayed by the effects of a secondary memory cache 103-1,if present. From the perspective of processors 102-2 . . . 102-P, oncethe data are latched onto the output pins of processor 102-1, all eightbytes of data are written to memory 104 in one contiguous, uninterruptedtransfer, which is enforced by the cache coherency protocol acrosssecondary memory caches 103-1 . . . 103-P, which may delay the write tomemory 104 if present.

However, if an 8-byte pointer value is written to a misaligned locationin memory 104, such as a memory address that crosses an 8-byte boundary,all eight bytes of data can not be transferred from processor 102-1using a single, 8-byte store instruction. Instead, processor 102-1 mayissue two separate and distinct store instructions. For example, if thememory address begins four bytes before an 8-byte boundary (e.g., 8N−4),the first store instruction transfers the four most significant bytes tomemory 104 (e.g., 8N−4), while the second store instruction transfersthe four least significant bytes to memory 104 (e.g., 8N). Importantly,between these two separate store instructions, processor 102-1 may beinterrupted, or, processor 102-1 may loose control of bus 101 to anothersystem component (e.g., processor 102-P, etc.). Consequently, thepointer value residing in memory 104 will be invalid until processor102-1 can complete the second store instruction. If another componentbegins a single, uninterruptible memory read to this memory location, aninvalid value will be returned as a presumably valid one.

Similarly, a new 4-byte pointer value may be written to a memory addressdivisible by four (e.g., 4N) using a single, uninterruptible operation.Note that in the example discussed above, a 4-byte pointer value may bewritten to the 8N−4 memory location using a single store instruction. Ofcourse, if a 4-byte pointer value is written to a location that crossesa 4-byte boundary, e.g., 4N−2, all four bytes of data can not betransferred from processor 102-1 using a single store instruction, andthe pointer value residing in memory 104 may be invalid for some periodof time.

System 100 may also include a read only memory (ROM) 106, or otherstatic storage device, coupled to bus 101 for storing static informationand instructions for processor 102-1. A storage device 108, such as amagnetic or optical disk, may be coupled to bus 101 for storinginformation and instructions. System 100 may also include display 110(e.g., an LCD monitor) and input device 112 (e.g., keyboard, mouse,trackball, etc.), coupled to bus 101. System 100 may include a pluralityof network interfaces 114-1 . . . 114-O, which may send and receiveelectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information. In an embodiment,network interface 114-1 may be coupled to bus 101 and local area network(LAN) 122, while network interface 114-O may coupled to bus 101 and widearea network (WAN) 124. Plurality of network interfaces 114-1 . . .114-O may support various network protocols, including, for example,Gigabit Ethernet (e.g., IEEE Standard 802.3-2002, published 2002), FiberChannel (e.g., ANSI Standard X.3230-1994, published 1994), etc.Plurality of network computers 120-1 . . . 120-N may be coupled to LAN122 and WAN 124. In one embodiment, LAN 122 and WAN 124 may bephysically distinct networks, while in another embodiment, LAN 122 andWAN 124 may be via a network gateway or router (not shown for clarity).Alternatively, LAN 122 and WAN 124 may be the same network.

As noted above, system 100 may provide DNS resolution services. In a DNSresolution embodiment, DNS resolution services may generally be dividedbetween network transport and data look-up functions. For example,system 100 may be a back-end look-up engine (LUE) optimized for datalook-up on large data sets, while plurality of network computers 120-1 .. . 120-N may be a plurality of front-end protocol engines (PEs)optimized for network processing and transport. The LUE may be apowerful multiprocessor server that stores the entire DNS record set inmemory 104 to facilitate high-speed, high-throughput searching andupdating. In an alternative embodiment, DNS resolution services may beprovided by a series of powerful multiprocessor servers, or LUEs, eachstoring a subset of the entire DNS record set in memory to facilitatehigh-speed, high-throughput searching and updating.

Conversely, the plurality of PEs may be generic, low profile, PC-basedmachines, running an efficient multitasking operating system (e.g., RedHat Linux® 6.2), that minimize the network processing transport load onthe LUE in order to maximize the available resources for DNS resolution.The PEs may handle the nuances of wire-line DNS protocol, respond toinvalid DNS queries and multiplex valid DNS queries to the LUE over LAN122. In an alternative embodiment including multiple LUEs storing DNSrecord subsets, the PEs may determine which LUE should receive eachvalid DNS query, and multiplex valid DNS queries to the appropriateLUEs. The number of PEs for a single LUE may be determined, for example,by the number of DNS queries to be processed per second and theperformance characteristics of the particular system. Other metrics mayalso be used to determine the appropriate mapping ratios and behaviors.

Generally, other large-volume, query-based embodiments may be supported,including, for example, telephone number resolution, SS7 signalingprocessing, geolocation determination, telephone number-to-subscribermapping, subscriber location and presence determination, etc.

In an embodiment, a central on-line transaction processing (OLTP) server140-1 may be coupled to WAN 124 and receive additions, modifications anddeletions (i.e., update traffic) to database 142-1 from various sources.OLTP server 140-1 may send updates to system 100, which includes a localcopy of database 142-1, over WAN 124. OLTP server 140-1 may be optimizedfor processing update traffic in various formats and protocols,including, for example, HyperText Transmission Protocol (HTTP), RegistryRegistrar Protocol (RRP), Extensible Provisioning Protocol (EPP),Service Management System/800 Mechanized Generic Interface (MGI), andother on-line provisioning protocols. A constellation of read-only LUEsmay be deployed in a hub and spoke architecture to provide high-speedsearch capability conjoined with high-volume, incremental updates fromOLTP server 140-1.

In an alternative embodiment, data may be distributed over multiple OLTPservers 140-1 . . . 140-S, each of which may be coupled to WAN 124. OLTPservers 140-1 . . . 140-S may receive additions, modifications, anddeletions (i.e., update traffic) to their respective databases 142-1 . .. 142-S (not shown for clarity) from various sources. OLTP servers 140-1. . . 140-S may send updates to system 100, which may include copies ofdatabases 142-1 . . . 142-S, other dynamically-created data, etc., overWAN 124. For example, in a geolocation embodiment, OLTP servers 140-1 .. . 140-S may receive update traffic from groups of remote sensors. Inanother alternative embodiment, plurality of network computers 120-1 . .. 120-N may also receive additions, modifications, and deletions (i.e.,update traffic) from various sources over WAN 124 or LAN 122. In thisembodiment, plurality of network computers 120-1 . . . 120-N may sendupdates, as well as queries, to system 100.

In the DNS resolution embodiment, each PE (e.g., each of the pluralityof network computers 120-1 . . . 120-N) may combine, or multiplex,several DNS query messages, received over a wide area network (e.g., WAN124), into a single Request SuperPacket and send the Request SuperPacketto the LUE (e.g., system 100) over a local area network (e.g., LAN 122).The LUE may combine, or multiplex, several DNS query message repliesinto a single Response SuperPacket and send the Response SuperPacket tothe appropriate PE over the local area network. Generally, the maximumsize of a Request or Response SuperPacket may be limited by the maximumtransmission unit (MTU) of the physical network layer (e.g., GigabitEthernet). For example, typical DNS query and reply message sizes ofless than 100 bytes and 200 bytes, respectively, allow for over 30queries to be multiplexed into a single Request SuperPacket, as well asover 15 replies to be multiplexed into a single Response SuperPacket.However, a smaller number of queries (e.g., 20 queries) may be includedin a single Request SuperPacket in order to avoid MTU overflow on theresponse (e.g., 10 replies). For larger MTU sizes, the number ofmultiplexed queries and replies may be increased accordingly.

Each multitasking PE may include an inbound thread and an outboundthread to manage DNS queries and replies, respectively. For example, theinbound thread may un-marshal the DNS query components from the incomingDNS query packets received over a wide area network and multiplexseveral milliseconds of queries into a single Request SuperPacket. Theinbound thread may then send the Request SuperPacket to the LUE over alocal area network. Conversely, the outbound thread may receive theResponse SuperPacket from the LUE, de-multiplex the replies containedtherein, and marshal the various fields into a valid DNS reply, whichmay then be transmitted over the wide area network. Generally, as notedabove, other large-volume, query-based embodiments may be supported.

In an embodiment, the Request SuperPacket may also include stateinformation associated with each DNS query, such as, for example, thesource address, the protocol type, etc. The LUE may include the stateinformation, and associated DNS replies, within the ResponseSuperPacket. Each PE may then construct and return valid DNS replymessages using the information transmitted from the LUE. Consequently,each PE may advantageously operate as a stateless machine, i.e., validDNS replies may be formed from the information contained in the ResponseSuperPacket. Generally, the LUE may return the Response SuperPacket tothe PE from which the incoming SuperPacket originated; however, othervariations may obviously be possible.

In an alternative embodiment, each PE may maintain the state informationassociated with each DNS query and include a reference, or handle, tothe state information within the Request SuperPacket. The LUE mayinclude the state information references, and associated DNS replies,within the Response SuperPacket. Each PE may then construct and returnvalid DNS reply messages using the state information referencestransmitted from the LUE, as well as the state information maintainedthereon. In this embodiment, the LUE may return the Response SuperPacketto the PE from which the incoming SuperPacket originated.

FIG. 2 is a detailed block diagram that illustrates a message datastructure, according to an embodiment of the present invention.Generally, message 200 may include header 210, having a plurality ofsequence number 211-1 . . . 211-S and a plurality of message counts212-1 . . . 212-S, and data payload 215.

In the DNS resolution embodiment, message 200 may be used for RequestSuperPackets and Response SuperPackets. For example, Request SuperPacket220 may include header 230, having a plurality of sequence number 231-1. . . 231-S and a plurality of message counts 232-1 . . . 232-S, anddata payload 235 having multiple DNS queries 236-1 . . . 236-Q,accumulated by a PE over a predetermined period of time, such as, forexample, several milliseconds. In one embodiment, each DNS query 236-1 .. . 236-Q may include state information, while in an alternativeembodiment, each DNS query 236-1 . . . 236-Q may include a handle tostate information.

Similarly, Response SuperPacket 240 may include header 250, having aplurality of sequence number 251-1 . . . 251-S and a plurality ofmessage counts 252-1 . . . 252-S, and data payload 255 having multipleDNS replies 256-1 . . . 256-R approximately corresponding to themultiple DNS queries contained within Request SuperPacket 220. In oneembodiment, each DNS reply 256-1 . . . 256-R may include stateinformation associated with the corresponding DNS query, while in analternative embodiment, each DNS reply 256-1 . . . 256-R may include ahandle to state information associated with the corresponding DNS query.Occasionally, the total size of the corresponding DNS replies may exceedthe size of data payload 255 of the Response SuperPacket 240. Thisoverflow may be limited, for example, to a single reply, i.e., the replyassociated with the last query contained within Request SuperPacket 220.Rather than sending an additional Response SuperPacket 240 containingonly the single reply, the overflow reply may be preferably included inthe next Response SuperPacket 240 corresponding to the next RequestSuperPacket. Advantageously, header 250 may include appropriateinformation to determine the extent of the overflow condition. Underpeak processing conditions, more than one reply may overflow into thenext Response SuperPacket.

For example, in Response SuperPacket 240, header 250 may include atleast two sequence numbers 251-1 and 251-2 and at least two messagecounts 252-1 and 252-2, grouped as two pairs of complementary fields.While there may be “S” number of sequence number and message countpairs, typically, S is a small number, such as, e.g., 2, 3, 4, etc.Thus, header 250 may include sequence number 251-1 paired with messagecount 252-1, sequence number 251-2 paired with message count 252-2, etc.Generally, message count 252-1 may reflect the number of repliescontained within data payload 255 that are associated with sequencenumber 251-1. In an embodiment, sequence number 251-1 may be a two-bytefield, while message count 252-1 may be a one-byte field.

In a more specific example, data payload 235 of Request SuperPacket 220may include seven DNS queries (as depicted in FIG. 2). In oneembodiment, sequence number 231-1 may be set to a unique value (e.g.,1024) and message count 232-1 may be set to seven, while sequence number231-2 and message count 232-2 may be set to zero. In another embodiment,header 230 may contain only one sequence number and one message count,e.g., sequence number 231-1 and message count 232-1 set to 1024 andseven, respectively. Typically, Request SuperPacket 220 may contain allof the queries associated with a particular sequence number.

Data payload 255 of Response SuperPacket 240 may include sevencorresponding DNS replies (as depicted in FIG. 2). In this example,header 250 may include information similar to Request SuperPacket 220,i.e., sequence number 251-1 set to the same unique value (i.e., 1024),message count 252-1 set to seven, and both sequence number 252-2 andmessage count 252-2 set to zero. However, in another example, datapayload 255 of Response SuperPacket 240 may include only fivecorresponding DNS replies, and message count 252-1 may be set to fiveinstead. The remaining two responses associated with sequence number1024 may be included within the next Response SuperPacket 240.

The next Request SuperPacket 240 may include a different sequence number(e.g., 1025) and at least one DNS query, so that the next ResponseSuperPacket 240 may include the two previous replies associated with the1024 sequence number, as well as at least one reply associated with the1025 sequence number. In this example, header 250 of the next ResponseSuperPacket 240 may include sequence number 251-1 set to 1024, messagecount 252-1 set to two, sequence number 251-2 set to 1025 and messagecount 252-2 set to one. Thus, Response SuperPacket 240 may include atotal of three replies associated with three queries contained withintwo different Request SuperPackets.

FIG. 3 is a detailed block diagram that illustrates a message latencydata structure architecture, according to an embodiment of the presentinvention. Message latency data structure 300 may include informationgenerally associated with the transmission and reception of message 200.In the DNS resolution embodiment, message latency data structure 300 mayinclude latency information about Request SuperPackets and ResponseSuperPackets; this latency information may be organized in a tableformat indexed according to sequence number value (e.g., index 301). Forexample, message latency data structure 300 may include a number of rowsN equal to the total number of unique sequence numbers, as illustrated,generally, by table elements 310, 320 and 330. In an embodiment,SuperPacket header sequence numbers may be two bytes in length anddefine a range of unique sequence numbers from zero to 2¹⁶−1 (i.e.,65,535). In this case, N may be equal to 65,536. Latency information mayinclude Request Timestamp 302, Request Query Count 303, ResponseTimestamp 304, Response Reply Count 305, and Response Message Count 306.In an alternative embodiment, latency information may also include anInitial Response Timestamp (not shown).

In an example, table element 320 illustrates latency information for aRequest SuperPacket 220 having a single sequence number 231-1 equal to1024. Request Timestamp 302 may indicate when this particular RequestSuperPacket was sent to the LUE. Request Query Count 303 may indicatehow many queries were contained within this particular RequestSuperPacket. Response Timestamp 304 may indicate when a ResponseSuperPacket having a sequence number equal to 1024 was received at thePE (e.g., network computer 120-N) and may be updated if more than oneResponse SuperPacket is received at the PE. Response Reply Count 305 mayindicate the total number of replies contained within all of thereceived Response SuperPackets associated with this sequence number(i.e., 1024). Response Message Count 306 may indicate how many ResponseSuperPackets having this sequence number (i.e., 1024) arrived at the PE.Replies to the queries contained within this particular RequestSuperPacket may be split over several Response SuperPackets, in whichcase, Response Timestamp 304, Response Reply Count 305, and ResponseMessage Count 306 may be updated as each of the additional ResponseSuperPackets are received. In an alternative embodiment, the InitialResponse Timestamp may indicate when the first Response SuperPacketcontaining replies for this sequence number (i.e., 1024) was received atthe PE. In this embodiment, Response Timestamp 304 may be updated whenadditional (i.e., second and subsequent) Response SuperPackets arereceived.

Various important latency metrics may be determined from the latencyinformation contained within message latency data structure 300. Forexample, simple cross-checking between Request Query Count 303 andResponse Reply Count 305 for a given index 301 (i.e., sequence number)may indicate a number of missing replies. This difference may indicatethe number of queries inexplicably dropped by the LUE. Comparing RequestTimestamp 302 and Response Timestamp 304 may indicate how well theparticular PE/LUE combination may be performing under the currentmessage load. The difference between the current Request SuperPacketsequence number and the current Response SuperPacket sequence number maybe associated with the response performance of the LUE; e.g., the largerthe difference, the slower the performance. The Response Message Count306 may indicate how many Response SuperPackets are being used for eachRequest SuperPacket, and may be important in DNS resolution trafficanalysis. As the latency of the queries and replies travelling betweenthe PEs and LUE increases, the PEs may reduce the number of DNS querypackets processed by the system.

Generally, the LUE may perform a multi-threaded look-up on the incoming,multiplexed Request SuperPackets, and may combine the replies intooutgoing, multiplexed Response SuperPackets. For example, the LUE mayspawn one search thread, or process, for each active PE and route allthe incoming Request SuperPackets from that PE to that search thread.The LUE may spawn a manager thread, or process, to control theassociation of PEs to search threads, as well as an update thread, orprocess, to update the database located in memory 104. Each searchthread may extract the search queries from the incoming RequestSuperPacket, execute the various searches, construct an outgoingResponse SuperPacket containing the search replies and send theSuperPacket to the appropriate PE. The update thread may receive updatesto the database, from OLTP 140-1, and incorporate the new data into thedatabase. In an alternative embodiment, plurality of network computers120-1 . . . 120-N may send updates to system 100. These updates may beincluded, for example, within the incoming Request SuperPacket messagestream.

Accordingly, by virtue of the SuperPacket protocol, the LUE may spendless than 15% of its processor capacity on network processing, therebydramatically increasing search query throughput. In an embodiment, anIBM® 8-way M80 may sustain search rates of 180 k to 220 k queries persecond (qps), while an IBM® 24-way S80 may sustain 400 k to 500 k qps.Doubling the search rates, i.e., to 500 k and 1M qps, respectively,simply requires twice as much hardware, i.e., e.g., two LUEs with theirattendant PEs. In another embodiment, a dual Pentium® III 866 MHzmulti-processor personal computer operating Red Hat Linux® 6.2 maysustain update rates on the order of 100 K/sec. Of course, increases inhardware performance also increase search and update rates associatedwith embodiments of the present invention, and as manufacturers replacethese multiprocessor computers with faster-performing machines, forexample, the sustained search and update rates may increasecommensurately. Generally, system 100 is not limited to a client orserver architecture, and embodiments of the present invention are notlimited to any specific combination of hardware and/or software.

FIG. 4 is a block diagram that illustrates a general databasearchitecture according to an embodiment of the present invention. Inthis embodiment, database 400 may include at least one table or group ofdatabase records 401, and at least one corresponding search index 402with pointers (indices, direct byte-offsets, etc.) to individual recordswithin the group of database records 401. For example, pointer 405 mayreference database record 410.

In one embodiment, database 400 may include at least one hash table 403as a search index with pointers (indices, direct byte-offsets, etc.)into the table or group of database records 401. A hash function may mapa search key to an integer value which may then be used as an index intohash table 403. Because more than one search key may map to a singleinteger value, hash buckets may be created using a singly-linked list ofhash chain pointers. For example, each entry within hash table 403 maycontain a pointer to the first element of a hash bucket, and eachelement of the hash bucket may contain a hash chain pointer to the nextelement, or database record, in the linked-list. Advantageously, a hashchain pointer may be required only for those elements, or databaserecords, that reference a subsequent element in the hash bucket.

Hash table 403 may include an array of 8-byte pointers to individualdatabase records 401. For example, hash pointer 404 within hash table403 may reference database record 420 as the first element within a hashbucket. Database record 420 may contain a hash chain pointer 424 whichmay reference the next element, or database record, in the hash bucket.Database record 420 may also include a data length 421 and associatedfixed or variable-length data 422. In an embodiment, a null character423, indicating the termination of data 422, may be included.Additionally, database record 420 may include a data pointer 425 whichmay reference another database record, either within the group ofdatabase records 401 or within a different table or group of databaserecords (not shown), in which additional data may be located.

System 100 may use various, well-known algorithms to search this datastructure architecture for a given search term or key. Generally,database 400 may be searched by multiple search processes, or threads,executing on at least one of the plurality of processors 102-1 . . .102-P. However, modifications to database 400 may not be integrallyperformed by an update thread (or threads) unless the search thread(s)are prevented from accessing database 400 for the period of timenecessary to add, modify, or delete information within database 400. Forexample, in order to modify database record 430 within database 400, thegroup of database records 401 may be locked by an update thread toprevent the search threads from accessing database 400 while the updatethread is modifying the information within database record 430. Thereare many well-known mechanisms for locking database 400 to preventsearch access, including the use of spin-locks, semaphores, mutexes,etc. Additionally, various off-the-shelf commercial databases providespecific commands to lock all or parts of database 400, e.g., the locktable command in the Oracle 8 Database, manufactured by OracleCorporation of Redwood Shores, Calif., etc.

FIG. 5 is a block diagram that illustrates a general databasearchitecture according to another embodiment of the present invention.In this embodiment, database 500 may include a highly-optimized,read-only, master snapshot file 510 and a growing, look-aside file 520.Master snapshot file 510 may include at least one table or group ofdatabase records 511, and at least one corresponding search index 512with pointers (indices, direct byte-offsets, etc.) to individual recordswithin the group of database records 511. Alternatively, master snapshotfile 510 may include at least one hash table 513 as a search index withpointers (indices, direct byte-offsets, etc.) into the table or group ofdatabase records 511. Similarly, look-aside file 520 may include atleast two tables or groups of database records, including databaseaddition records 521 and database deletion records 531. Correspondingsearch indices 522 and 532 may be provided, with pointers (indices,direct byte-offsets, etc.) to individual records within the databaseaddition records 521 and database deletion records 531. Alternatively,look-aside file 520 may include hash tables 523 and 533 as searchindices, with pointers (indices, direct byte-offsets, etc.) intodatabase addition records 521 and database deletion records 531,respectively.

System 100 may use various, well-known algorithms to search this datastructure architecture for a given search term or key. In a typicalexample, look-aside file 520 may include all the recent changes to thedata, and may be searched before read-only master snapshot file 510. Ifthe search key is found in look-aside file 520, the response is returnedwithout accessing snapshot file 510, but if the key is not found, thensnapshot file 510 may be searched. However, when look-aside file 520 nolonger fits in memory 104 with snapshot file 510, search query ratesdrop dramatically, by a factor of 10 to 50, or more, for example.Consequently, to avoid or minimize any drop in search query rates,snapshot file 510 may be periodically updated, or recreated, byincorporating all of the additions, deletions and modificationscontained within look-aside file 520

Data within snapshot file 510 are not physically altered but logicallyadded, modified or deleted. For example, data within snapshot file 510may be deleted, or logically “forgotten,” by creating a correspondingdelete record within database deletion records 531 and writing a pointerto the delete record to the appropriate location in hash table 533. Datawithin snapshot file 510 may be logically modified by copying a datarecord from snapshot file 510 to a new data record within databaseaddition records 521, modifying the data within the new entry, and thenwriting a pointer to the new entry to the appropriate hash table (e.g.,hash table 522) or chain pointer within database addition records 521.Similarly, data within snapshot file 510 may be logically added tosnapshot file 510 by creating a new data record within database additionrecords 521 and then writing a pointer to the new entry to theappropriate hash table (e.g., hash table 522) or chain pointer withindatabase addition records 521.

In the DNS resolution embodiment, for example, snapshot file 510 mayinclude domain name data and name server data, organized as separatedata tables, or blocks, with separate search indices (e.g., 511-1,511-2, 512-1, 512-2, 513-1, 513-2, etc., not shown for clarity).Similarly, look-aside file 520 may include additions and modificationsto both the domain name data and the name server data, as well asdeletions to both the domain name data and the name server data (e.g.,521-1, 521-2, 522-1, 522-2, 523-1, 523-2, 531-1, 531-2, 532-1, 532-2,533-1, 533-2, etc., not shown for clarity).

FIG. 6 is a detailed block diagram that illustrates a non-concurrencycontrolled data structure architecture, according to an embodiment ofthe present invention. Generally, database 600 may be organized into asingle, searchable representation of the data. Data set updates may becontinuously incorporated into database 600, and deletes ormodifications may be physically performed on the relevant databaserecords to free space within memory 104, for example, for subsequentadditions or modifications. The single, searchable representation scalesextremely well to large data set sizes and high search and update rates,and obviates the need to periodically recreate, propagate and reloadsnapshot files among multiple search engine computers.

In a DNS resolution embodiment, for example, database 600 may includedomain name data 610 and name server data 620. Domain name data 610 andname server data 620 may include search indices with pointers (indices,direct byte-offsets, etc.) into blocks of variable length records. Asdiscussed above, a hash function may map a search key to an integervalue which may then be used as an index into a hash table. Similarly,hash buckets may be created for each hash table index using asingly-linked list of hash chain pointers. Domain name data 610 mayinclude, for example, a hash table 612 as a search index and a block ofvariable-length domain name records 611. Hash table 612 may include anarray of 8-byte pointers to individual domain name records 611, such as,for example, pointer 613 referencing domain name record 620.Variable-length domain name record 620 may include, for example, a nextrecord offset 621, a name length 622, a normalized name 623, a chainpointer 624 (i.e., e.g., pointing to the next record in the hash chain),a number of name servers 625, and a name server pointer 626. The size ofboth chain pointer 624 and name server pointer 626 may be optimized toreflect the required block size for each particular type of data, e.g.,eight bytes for chain pointer 624 and four bytes for name server pointer626.

Name server data 630 may include, for example, a hash table 632 as asearch index and a block of variable-length name server records 631.Hash table 632 may include an array of 4-byte pointers to individualname server records 631, such as, for example, pointer 633 referencingname server record 640. Variable-length name server record 640 mayinclude, for example, a next record offset 641, a name length 642, anormalized name 643, a chain pointer 644 (i.e., e.g., pointing to thenext record in the hash chain), a number of name server networkaddresses 645, a name server address length 646, and a name servernetwork address 647, which may be, for example, an Internet Protocol(IP) network address. Generally, name server network addresses may bestored in ASCII (American Standard Code for Information Interchange,e.g., ISO-14962-1997, ANSI-X3.4-1997, etc.) or binary format; in thisexample, name server network address length 646 indicates that nameserver network address 647 is stored in binary format (i.e., fourbytes). The size of chain pointer 644 may also be optimized to reflectthe required name server data block size, e.g., four bytes.

Generally, both search indices, such as hash tables, and variable-lengthdata records may be structured so that 8-byte pointers are located on8-byte boundaries in memory. For example, hash table 612 may contain acontiguous array of 8-byte pointers to domain name records 611, and maybe stored at a memory address divisible by eight (i.e., an 8-byteboundary, or 8N). Similarly, both search indices, such as hash tables,and variable-length data records may be structured so that 4-bytepointers are located on 4-byte boundaries in memory. For example, hashtable 632 may contain a contiguous array of 4-byte pointers to nameserver records 631, and may be stored at a memory address divisible byfour (i.e., a 4-byte boundary, or 4N). Consequently, modifications todatabase 600 may conclude by updating a pointer to an aligned address inmemory using a single uninterruptible operation, including, for examplewriting a new pointer to the search index, such as a hash table, orwriting a new hash chain pointer to a variable-length data record.

FIG. 7 is a detailed block diagram that illustrates a non-concurrencycontrolled data structure architecture, according to an embodiment ofthe present invention. Generally, database 700 may also be organizedinto a single, searchable representation of the data. Data set updatesmay be continuously incorporated into database 700, and deletes ormodifications may be physically performed on the relevant databaserecords to free space within memory 104, for example, for subsequentadditions or modifications. The single, searchable representation scalesextremely well to large data set sizes and high search and update rates,and obviates the need to periodically recreate, propagate and reloadsnapshot files among multiple search engine computers.

Many different physical data structure organizations are possible. Anexemplary organization may use an alternative search index to hashtables for ordered, sequential access to the data records, such as theternary search tree (trie), or TST, which combines the features ofbinary search trees and digital search tries. In a text-basedapplications, such as, for example, whois, domain name resolution usingDNS Secure Extensions (Internet Engineering Taskforce Request forComments: 2535), etc. TSTs advantageously minimize the number ofcomparison operations required to be performed, particularly in the caseof a search miss, and may yield search performance metrics exceedingsearch engine implementations with hashing. Additionally, TSTs may alsoprovide advanced text search features, such as, e.g., wildcard searches,which may be useful in text search applications, such as, for example,whois, domain name resolution, Internet content search, etc.

In an embodiment, a TST may contain a sequence of nodes linked togetherin a hierarchical relationship. A root node may be located at the top ofthe tree, related child nodes and links may form branches, and leafnodes may terminate the end of each branch. Each leaf node may beassociated with a particular search key, and each node on the path tothe leaf node may contain a single, sequential element of the key. Eachnode in the tree contains a comparison character, or split value, andthree pointers to other successive, or “child,” nodes in the tree. Thesepointers reference child nodes whose split values are less than, equalto, or greater than the node's split value. Searching the TST for aparticular key, therefore, involves traversing the tree from the rootnode to a final leaf node, sequentially comparing each element, orcharacter position, of the key with the split values of the nodes alongthe path. Additionally, a leaf node may also contain a pointer to a keyrecord, which may, in turn, contain at least one pointer to a terminaldata record containing the record data associated with the key (e.g., anIP address). Alternatively, the key record may contain the record datain its entirety. Record data may be stored in binary format, ASCII textformat, etc.

In an embodiment, database 700 may be organized as a TST, including aplurality of fixed-length search nodes 701, a plurality ofvariable-length key data records 702 and a plurality of variable-lengthterminal data records 703. Search nodes 701 may include various types ofinformation as described above, including, for example, a comparisoncharacter (or value) and position, branch node pointers and a keypointer. The size of the node pointers may generally be determined bythe number of nodes, while the size of the key pointers may generally bedetermined by the size of the variable-length key data set. Key datarecords 702 may contain key information and terminal data information,including, for example, pointers to terminal data records or embeddedrecord data, while terminal data records 703 may contain record data.

In an embodiment, each fixed-length search node may be 24 bytes inlength. Search node 710, for example, may contain an eight-bitcomparison character (or byte value) 711, a 12-bit character (or byte)position 712, and a 12-bit node type/status (not shown for clarity);these data may be encoded within the first four bytes of the node. Thecomparison character 711 may be encoded within the first byte of thenode as depicted in FIG. 7, or, alternatively, character position 712may be encoded within the first 12 bits of the node in order to optimizeaccess to character position 712 using a simple shift operation. Thenext 12 bytes of each search node may contain three 32-bit pointers,i.e., pointer 713, pointer 714 and pointer 715, representing “lessthan,” “equal to,” and “greater than” branch node pointers,respectively. These pointers may contain a counter, or node index,rather than a byte-offset or memory address. For fixed-length searchnodes, the byte-offset may be calculated from the counter, or indexvalue, and the fixed-length, e.g., counter*length. The final four bytesmay contain a 40-bit key pointer 716, which may be a null valueindicating that a corresponding key data record does not exist (shown)or a pointer to an existing corresponding key data record (not shown),as well as other data, including, for example, a 12-bit key length and a12-bit pointer type/status field. Key pointer 716 may contain a byteoffset to the appropriate key data record, while the key length may beused to optimize search and insertion when eliminating one-way branchingwithin the TST. The pointer type/status field may contain informationused in validity checking and allocation data used in memory management.

In an embodiment, key data record 750 may include, for example, avariable-length key 753 and at least one terminal data pointer. Asdepicted in FIG. 7, key data record 750 includes two terminal datapointers: terminal data pointer 757 and terminal data pointer 758. Keydata record 750 may be prefixed with a 12-bit key length 751 and a12-bit terminal pointer count/status 752, and may include padding (notshown for clarity) to align the terminal data pointer 757 and terminaldata pointer 758 on an eight-byte boundary in memory 104. Terminal datapointer 757 and terminal data pointer 758 may each contain various data,such as, for example, terminal data type, length, status or data usefulin binary record searches. Terminal data pointer 757 and terminal datapointer 758 may be sorted by terminal data type for quicker retrieval ofspecific resource records (e.g., terminal data record 760 and terminaldata record 770). In another embodiment, key data record 740 may includeembedded terminal data 746 rather than, or in addition to, terminal datarecord pointers. For example, key data record 740 may include a keylength 741, a terminal pointer count 742, a variable-length key 743, thenumber of embedded record elements 744, followed by a record elementlength 745 (in bytes, for example) and embedded record data 746 (e.g., astring, a byte sequence, etc.) for each of the number of embedded recordelements 744.

In an embodiment, terminal data record 760, for example, may include a12-bit length 761, a 4-bit status, and a variable-length string 762(e.g., an IP address). Alternatively, variable length string 762 may bea byte sequence. Terminal data record 760 may include padding to aligneach terminal data record to an 8-byte boundary in memory 104.Alternatively, terminal data record 760 may include padding to a 4-byteboundary, or, terminal data record 760 may not include any padding.Memory management algorithms may determine, generally, whether terminaldata records 760 are padded to 8-byte, 4-byte, or 0-byte boundaries.Similarly, terminal data record 770 may include a 12-bit length 771, a4-bit status, and a variable-length string 772 (e.g., an IP address).

Generally, both search indices, such as TSTs, and data records may bestructured so that 8-byte pointers are located on 8-byte boundaries inmemory. For example, key pointer 726 may contain an 8-byte (or less)pointer to key data record 740, and may be stored at a memory addressdivisible by eight (i.e., an 8-byte boundary, or 8N). Similarly, bothsearch indices, such as TSTs, and data records may be structured so that4-byte pointers are located on 4-byte boundaries in memory. For example,node branch pointer 724 may contain a 4-byte (or less) pointer to node730, and may be stored at a memory address divisible by four (i.e., a4-byte boundary, or 4N). Consequently, modifications to database 700 mayconclude by updating a pointer to an aligned address in memory using asingle uninterruptible operation, including, for example writing a newpointer to the search index, such as a TST node, or writing a newpointer to a data record.

FIG. 8 is a detailed block diagram that illustrates another datastructure architecture, according to an embodiment of the presentinvention. As above, database 800 may also be organized into a single,searchable representation of the data. Data set updates may becontinuously incorporated into database 800, and deletes ormodifications may be physically performed on the relevant databaserecords to free space within memory 104, for example, for subsequentadditions or modifications. The single, searchable representation scalesextremely well to large data set sizes and high search and update rates,and obviates the need to periodically recreate, propagate and reloadsnapshot files among multiple search engine computers.

Other search index structures are possible for accessing record data, Inan embodiment, database 800 may use an alternative ordered search index,organized as an ordered access key tree (i.e., “OAK tree”). Database 800may include, for example, a plurality of variable-length search nodes801, a plurality of variable-length key records 802 and a plurality ofvariable-length terminal data records 803. Search nodes 801 may includevarious types of information as described above, such as, for example,search keys, pointers to other search nodes, pointers to key records,etc. In an embodiment, plurality of search nodes 801 may includevertical and horizontal nodes containing fragments of search keys (e.g.,strings), as well as pointers to other search nodes or key records.Vertical nodes may include, for example, at least one search key, orcharacter, pointers to horizontal nodes within the plurality of searchnodes 801, pointers to key records within the plurality of key records802, etc. Horizontal nodes may include, for example, at least two searchkeys, or characters, pointers to vertical nodes within the plurality ofsearch nodes 801, pointers to horizontal nodes within the plurality ofsearch nodes 801, pointers to key records within the plurality of keyrecords 802, etc. Generally, vertical nodes may include a sequence ofkeys (e.g., characters) representing a search key fragment (e.g.,string), while horizontal nodes may include various keys (e.g.,characters) that may exist at a particular position within the searchkey fragment (e.g., string).

In an embodiment, plurality of search nodes 801 may include verticalnode 810, vertical node 820 and horizontal node 830. Vertical node 810may include, for example, a 2-bit node type 811 (e.g., “10”), a 38-bitaddress 812, an 8-bit length 813 (e.g., “8”), an 8-bit first character814 (e.g., “I”) and an 8-bit second character 815 (e.g., “null”). Inthis example, address 812 may point to the next node in the search tree,i.e., vertical node 820. In an embodiment, 38-bit address 812 mayinclude a 1-bit terminal/nodal indicator and a 37-bit offset address toreference one of the 8-byte words within a 1 Tbyte (˜10¹² byte) addressspace of memory 104. Accordingly, vertical node 810 may be eight bytes(64 bits) in length, and, advantageously, may be located on an 8-byteword boundary within memory 104. Generally, each vertical node withinplurality of search nodes 801 may be located on an 8-byte word boundarywithin memory 104.

A vertical node may include a multi-character, search key fragment(e.g., string). Generally, search keys without associated key datarecords may be collapsed into a single vertical node to effectivelyreduce the number of vertical nodes required within plurality of searchnodes 801. In an embodiment, vertical node 810 may include eight bitsfor each additional character, above two characters, within the searchkey fragment, such as, for example, 8-bit characters 816-1, 816-2 . . .816-N (shown in phantom outline). Advantageously, vertical node 810 maybe padded to a 64-bit boundary within memory 104 in accordance with thenumber of additional characters located within the string fragment. Forexample, if nine characters are to be included within vertical node 810,then characters one and two may be assigned to first character 814 andsecond character 815, respectively, and 56 bits of additional characterinformation, corresponding to characters three through nine, may beappended to vertical node 810. An additional eight bits of padding maybe included to align the additional character information on an 8-byteword boundary.

Similarly, vertical node 820 may include, for example, a 2-bit node type821 (e.g., “10”), a 38-bit address 822, an 8-bit length 823 (e.g., “8”),an 8-bit first character 824 (e.g., “a”) and an 8-bit second character825 (e.g., “null”). In this example, address 822 may point to the nextnode in the search tree, i.e., horizontal node 830. Accordingly,vertical node 820 may be eight bytes in length, and, advantageously, maybe located on an 8-byte word boundary within memory 104. Of course,additional information may also be included within vertical node 820 ifrequired, as described above with reference to vertical node 810.

Horizontal node 830 may include, for example, a 2-bit node type 831(e.g., “01”), a 38-bit first address 832, an 8-bit address count 833(e.g., 2), an 8-bit first character 834 (e.g., “•”), an 8-bit lastcharacter 835 (e.g., “w”), a variable-length bitmap 836 and a 38-bitsecond address 837. In this example, first character 834 may include asingle character, “•” representing the search key fragment “la” definedby vertical nodes 810 and 820, while last character 831 may include asingle character “w,” representing the search key fragment “law” definedby vertical nodes 810 and 820, and the last character 835 of horizontalnode 830. First address 832 may point to key data record 840, associatedwith the search key fragment “la,” while second address 837 may point tokey data record 850 associated with the search key fragment “law.”

Bitmap 836 may advantageously indicate which keys (e.g., characters) arereferenced by horizontal node 830. A “1” within a bit position in bitmap836 indicates that the key, or character, is referenced by horizontalnode 830, while a “0” within a bit position in bitmap 836 may indicatethat the key, or character, is not referenced by horizontal node 830.Generally, the length of bitmap 836 may depend upon the number ofsequential keys, or characters, between first character 834 and lastcharacter 835, inclusive of these boundary characters. For example, iffirst character 834 is “a” and last character 835 is “z,” then bitmap836 may be 26 bits in length, where each bit corresponds to one of thecharacters between, and including, “a” through “z.” In this example,additional 38-bit addresses would be appended to the end of horizontalnode 830, corresponding to each of the characters represented withinbitmap 836. Each of these 38-bit addresses, as well as bitmap 836, maybe padded to align each quantity on an 8-byte word boundary withinmemory 104. In an embodiment, the eight-bit ASCII character set may beused as the search key space so that bitmap 836 may be as long as 256bits (i.e., 2⁸ bits or 32 bytes). In the example depicted in FIG. 8, dueto the special reference character “•” and address count 833 of “2,”bitmap 836 may be two bits in length and may include a “1” in each bitposition corresponding to last character 835.

In an embodiment, and as discussed with reference to key data record 750(FIG. 7), key data record 850 may include, for example, avariable-length key 853 and at least one terminal data pointer. Asdepicted in FIG. 8, key data record 850 includes two terminal datapointers, terminal data pointer 857 and terminal data pointer 858. Keydata record 850 may be prefixed with a 12-bit key length 851 and a12-bit terminal pointer count/status 852, and may include padding (notshown for clarity) to align the terminal data pointer 857 and terminaldata pointer 858 on an 8-byte boundary in memory 104. Terminal datapointer 857 and terminal data pointer 858 may each contain a 10-bitterminal data type and other data, such as, for example, length, statusor data useful in binary record searches. Terminal data pointer 857 andterminal data pointer 858 may be sorted by terminal data type forquicker retrieval of specific resource records (e.g., terminal datarecord 860 and terminal data record 870).

In another embodiment, and as discussed with reference to key datarecord 740 (FIG. 7), key data record 840 may include embedded terminaldata 846 rather than a terminal data record pointer. For example, keydata record 840 may include a key length 841, a terminal pointer count842, a variable-length key 843, the number of embedded record elements844, followed by a record element length 845 (in bytes, for example) andembedded record data 846 (e.g., a string, a byte sequence, etc.) foreach of the number of embedded record elements 844.

In another embodiment, and as discussed with reference to terminal datarecord 760 (FIG. 7), terminal data record 860, for example, may includea 12-bit length 861, a 4-bit status, and a variable-length string 862(e.g., an IP address). Alternatively, variable length string 862 may bea byte sequence. Terminal data record 860 may include padding (not shownfor clarity) to align each terminal data record to an 8-byte boundary inmemory 104. Alternatively, terminal data record 860 may include padding(not shown for clarity) to a 4-byte boundary, or, terminal data record860 may not include any padding. Memory management algorithms maydetermine, generally, whether terminal data records 760 are padded to8-byte, 4-byte, or 0-byte boundaries. Similarly, terminal data record870 may include a 12-bit length 871, a 4-bit status, and avariable-length string 872 (e.g., an IP address).

Generally, both search indices, such as OAK trees, and data records maybe structured so that 8-byte pointers are located on 8-byte boundariesin memory. For example, vertical node 810 may contain an 8-byte (orless) pointer to vertical node 820, and may be stored at a memoryaddress divisible by eight (i.e., an 8-byte boundary, or 8N). Similarly,both search indices, such as OAK trees, and data records may bestructured so that 4-byte pointers are located on 4-byte boundaries inmemory. Consequently, modifications to database 800 may conclude byupdating a pointer to an aligned address in memory using a singleuninterruptible operation, including, for example writing a new pointerto the search index, such as an OAK trees node, or writing a new pointerto a data record.

The various embodiments discussed above with reference to FIG. 8 presentmany advantages. For example, an OAK tree data structure is extremelyspace efficient and 8-bit clean. Regular expression searches may be usedto search vertical nodes containing multi-character string fragments,since the 8-bit first character (e.g., first character 814), the 8-bitsecond character (e.g., second character 8-15) and any additional 8-bitcharacters (e.g., additional characters 816-1 . . . 816-N) may becontiguously located within the vertical node (e.g., vertical node 810).Search misses may be discovered quickly, and, no more than N nodes mayneed to be traversed to search for an N-character length search string.

FIG. 9 is a top level flow diagram that illustrates a method forsearching and concurrently updating a database without the use ofdatabase locks or access controls, according to embodiments of thepresent invention.

An update thread and a plurality of search threads may be created (900).In an embodiment, system 100 may spawn a single update thread toincorporate updates to the local database received, for example, fromOLTP server 140-1 over WAN 124. In other embodiments, system 100 mayreceive updates from OLTP servers 140-1 . . . 140-S over WAN 124, andfrom plurality of network computers 120-1 . . . 120-N over WAN 124 orLAN 122. System 100 may also spawn a search thread in response to eachsession request received from the plurality of network computers 120-1 .. . 120-N. For example, a manger thread may poll one or more controlports, associated with one or more network interfaces 114-1 . . . 114-O,for session requests transmitted from the plurality of network computers120-1 . . . 120-N. Once a session request from a particular networkcomputer 120-1 . . . 120-N is received, the manage thread may spawn asearch thread and associate the search thread with that particularnetwork computer (e.g., PE).

In an alternative embodiment, system 100 may spawn a number of searchthreads without polling for session requests from the plurality ofnetwork computers 120-1 . . . 120-N. In this embodiment, the searchthreads may not be associated with particular network computers and maybe distributed evenly among the plurality of processors 102-1 . . .102-P. Alternatively, the search threads may execute on a subset of theplurality of processors 102-1 . . . 102-P. The number of search threadsmay not necessarily match the number of network computers (e.g., N).

A plurality of search queries may be received (910) over the network. Inan embodiment, plurality of network computers 120-1 . . . 120-N may sendthe plurality of search queries to system 100 over LAN 122, or,alternatively, WAN 124. The plurality of search queries may contain, forexample, a search term or key, as well as state information that may beassociated with each query (e.g., query source address, protocol type,etc.). State information may be explicitly maintained by system 100, or,alternatively, a state information handle may be provided. In apreferred embodiment, each of the plurality of network computers 120-1 .. . 120-N may multiplex a predetermined number of search queries into asingle network packet for transmission to system 100 (e.g., a RequestSuperPacket 220 as depicted in FIG. 2).

Each search query may be assigned (920) to one of the search threads forprocessing. In an embodiment, each search thread may be associated withone of the plurality of network computers 120-1 . . . 120-N and all ofthe search queries received from that particular network computer may beassigned (920) to the search thread. In other words, one search threadmay process all of the search queries arriving from a single networkcomputer (e.g., a single PE). In an embodiment, each search thread mayextract individual search queries from a single, multiplexed networkpacket (e.g., Request SuperPacket 220 as depicted in FIG. 2), or,alternatively, the extraction may be performed by a different process orthread.

In another embodiment, the search queries received from each of theplurality of network computers 120-1 . . . 120-N may be assigned (920)to different search threads. In this embodiment, the multi-threadassignment may be based on an optimal distribution function which mayincorporate various system parameters including, for example, processorloading. Of course, the assignment of search queries to search threadsmay change over time, based upon various system parameters, includingprocessor availability, system component performance, etc. Variousmechanisms may be used to convey search queries to assigned searchthreads within system 100, such as, for example, shared memory,inter-process messages, tokens, semaphores, etc.

Each search thread may search (930) the database based on the assignedsearch queries. Searching the database may depend upon the underlyingstructure of the database.

Referring to the database embodiment illustrated in FIG. 4, database 400may be searched (930) for the search key. The data record (e.g.,database record 420) corresponding to the search key may then bedetermined. Referring to the database embodiment illustrated in FIG. 5,look-aside file 520 may first be searched (930) for the search key, and,if a match is not determined, then snapshot file 510 may be searched(930). The data record corresponding to the search key may then bedetermined.

Referring to the database embodiment illustrated in FIG. 6, domain namedata 610 may first be searched (930) for the search key, and then theresource data within name server data 630, corresponding to the searchkey, may then be determined. For example, for the “la.com” search key, amatch may be determined with domain name record 620 in domain name data610. The appropriate information may be extracted, including, forexample, name server pointer 626. Then, the appropriate name serverrecord 640 may be indexed using name server pointer 626, and name servernetwork address 647 may be extracted.

Referring to the database embodiment illustrated in FIG. 7, the TST maybe searched (930) for the search key, from which the resource data maybe determined. For example, for the “law.com” search key, search nodes701 may be searched (930), and a match determined with node 730. Keypointer 736 may be extracted, from which the key data record 750 may bedetermined. The number of terminal data pointers 752 may then beidentified and each terminal data pointer may be extracted. For example,terminal data pointer 757 may reference terminal data record 760 andterminal data pointer 758 may reference and terminal data record 770.The variable-length resource data, e.g., name server network address 762and name server network address 772, may then be extracted from eachterminal data record using the length 761 and 771, respectively.

Referring to the database embodiment illustrated in FIG. 8, the OAK treemay be searched (930) for the search key, from which the resource datamay be determined. For example, for the “law.com” search key, searchnodes 801 may be searched (930), and a match determined with node 830.Second address 837 may be extracted, from which the key data record 850may be determined. The number of terminal data pointers 852 may then beidentified and each terminal data pointer may be extracted. For example,terminal data pointer 857 may reference terminal data record 860 andterminal data pointer 858 may reference and terminal data record 870.The variable-length resource data, e.g., name server network address 862and name server network address 872, may then be extracted from eachterminal data record using the length 861 and 871, respectively.

Each search thread may create (940) a plurality of search repliescorresponding to the assigned search queries. If a match is not foundfor a particular search key, the reply may include an appropriateindication, such as, for example the null character. For domain nameresolution, for example, a search key might be “law.com” and thecorresponding resource data might be “180.1.1.1”. More than one nameserver network address may be associated with a search key, in whichcase, more than one name server network address may be determined.

The replies may be sent (950) over the network. In an embodiment, eachsearch thread may multiplex the appropriate replies into a singlenetwork packet (e.g., Response SuperPacket 240) corresponding to thesingle network packet containing the original queries (e.g., RequestSuperPacket 220). Alternatively, a different process or thread maymultiplex the appropriate replies into the single network packet. Theresponse network packet may then be sent (950) to the appropriatenetwork computer within the plurality of network computers 120-1 . . .120-N via LAN 122, or alternatively, WAN 124. In one embodiment, theresponse packets may be sent to the same network computer from which therequest packets originated, while in another embodiment, the responsepackets may be sent to a different network computer.

The update thread may receive (960) new information over the network. Inan embodiment, new information may be sent, for example, from OLTPserver 140-1 to system 100 over WAN 124. In other embodiments, system100 may receive updates from OLTP servers 140-1 . . . 140-S over WAN124, and from plurality of network computers 120-1 . . . 120-N over WAN124 or LAN 122. In the DNS resolution embodiment, for example, the newinformation may include new domain name data, new name server data, anew name server for an existing domain name, etc. Alternatively, the newinformation may indicate that a domain name, name server, name servernetwork address, etc., may be deleted from the database. Generally, anyinformation contained within the database may be added, modified ordeleted, as appropriate.

The update thread may create (970) a new element in the databasecontaining the new information. Generally, modifications to informationcontained within an existing element of the database are incorporated bycreating a new element based on the existing element and then modifyingthe new element to include the new information. During this process, thenew element may not be visible to the search threads or processescurrently executing on system 100 until the new element has beencommitted to the database. Generally, additions to the database may beaccomplished in a similar fashion, without necessarily using informationcontained within an existing element. In one embodiment, the deletion ofan existing element from the database may be accomplished by adding anew, explicit “delete” element to the database. In another embodiment,the deletion of an existing element from the database may beaccomplished by overwriting a pointer to the existing element with anappropriate indicator (e.g., a null pointer, etc.). In this embodiment,the update thread does not create a new element in the databasecontaining new information.

In the DNS resolution embodiment, for example, the new information mayinclude a new domain name to be added to the database. In this example,for simplicity, the new domain name may reference an existing nameserver. Referring to FIG. 6, memory space for a new domain name record615 may be allocated from a memory pool associated with the domain namerecords 611, or, alternatively, from a general memory pool associatedwith domain name data 610. The new domain name may be normalized andcopied to the new domain name record 615, and a pointer to an existingname server (e.g., name server record 655) may be determined and copiedto the new domain name record 615. Other information may be calculatedand added to new domain name record 615, such as, for example, a numberof name servers, a chain pointer, etc. In more complicated examples, thenew information may include a new search key with corresponding resourcedata.

Referring to FIG. 7, a new search node 705, as well as a new key datarecord 780, may first be created. In this example, the new search node705 may include a comparison character (“m”), in the first position,that is greater than the comparison character (“I”), in the firstposition, of existing search node 710. Consequently, search node 705 maybe inserted in the TST at the same “level” (i.e., 1^(st) characterposition) as search node 710. Before search node 705 is committed to thedatabase, the 4-byte “greater than” pointer 715 of search node 710 maycontain a “null” pointer. Search node 705 may also include a 4-byte keypointer 706 which may contain a 40-bit pointer to the new key datarecord 780. Key data record 780 may include a key length 781 (e.g., “5”)and type 782 (e.g., indicating embedded resource data), a variablelength key 783 (e.g., “m.com”), a number of embedded resources 784(e.g., “1”), a resource length 785 (e.g., “9”), and a variable-lengthresource string 786 or byte sequence (e.g., “180.1.1.1”). In anembodiment, memory space may be allocated for search node 705 from amemory pool associated with TST nodes 701, while memory space may beallocated for the key data record 770 from a memory pool associated withplurality of key data records 702.

Referring to FIG. 8, a new search node 890, as well as a new key datarecord 880, may first be created. In this example, the new search node890 may be a horizontal node including, for example, a 2-bit node type891 (e.g., “01”), a 38-bit first address 892, an 8-bit address count 893(e.g., 2), an 8-bit first character 894 (e.g., “I”), an 8-bit lastcharacter 895 (e.g., “m”), a variable-length bitmap 896 and a 38-bitsecond address 897. First address 892 may point to vertical node 820,the next vertical node in the “I . . .” search string path, while secondaddress 897 may point to key data record 880 associated with the searchkey fragment “m.” Key data record 880 may include a key length 881(e.g., “5”) and type 882 (e.g., indicating embedded resource data), avariable length key 883 (e.g., “m.com”), a number of embedded resources884 (e.g., “1”), a resource length 885 (e.g., “9”), and avariable-length resource string 886 or byte sequence (e.g.,“180.1.1.1”). In an embodiment, memory space may be allocated for searchnode 890 from a memory pool associated with plurality of search nodes801, while memory space may be allocated for key data record 880 from amemory pool associated with plurality of key data records 802.

The update thread may write (980) a pointer to the database using asingle uninterruptible operation. Generally, a new element may becommitted to the database, (i.e., become visible to the search threads,or processes), instantaneously by writing a pointer to the new elementto the appropriate location within the database. As discussed above, theappropriate location may be aligned in memory, so that the singleoperation includes a single store instruction of an appropriate length.In an embodiment, an existing element may be deleted from the database(i.e., become invisible to the search threads, or processes) byinstantaneously overwriting a pointer to the existing element with anappropriate indicator (e.g., a “null” pointer, etc.). Again, theappropriate location may be aligned in memory, so that the singleoperation includes a single store instruction of an appropriate length.

Referring to FIG. 6, an 8-byte pointer corresponding to domain namerecord 620 may be written to hash table 612 (e.g., element 613)Importantly, the hash table entries are aligned on 8-byte boundaries inmemory 104 to ensure that a single, 8-byte store instruction is used toupdate this value. Referring to FIG. 7, a 4-byte pointer correspondingto the new search node 705 may be written to the 4-byte “greater-than”node pointer 715 within search node 710. Importantly, the node pointer715 is aligned on a 4-byte boundary in memory 104 to ensure that asingle, 4-byte store instruction may be used to update this value.Referring to FIG. 8, plurality of search nodes 801 may also include atop-of-tree address 899, which may be aligned on an 8-byte word boundaryin memory 104 and reference the first node within plurality of searchnodes 801 (i.e., e.g., vertical node 810). An 8-byte pointercorresponding to the new search node 890 may be written to thetop-of-tree address 899 using a single store instruction. In each ofthese embodiments, just prior to the store instruction, the new data arenot visible to the search threads, while just after the storeinstruction, the new data are visible to the search threads. Thus, witha single, uninterruptible operation, the new data may be committed tothe database without the use of database locks or access controls.

In an embodiment, the update thread may physically delete (990) anexisting element after the pointer is written (980) to the database.Advantageously, for existing elements of the database that are modifiedor deleted, the physical deletion of these elements from memory 104 maybe delayed to preserve consistency of in-progress searches. For example,after an existing element has been modified and the corresponding newelement committed to the database, the physical deletion of the existingelement from memory 104 may be delayed so that existing search threadsthat have a result, acquired just before the new element was committedto the database, may continue to use the previous state of the data.Similarly, after an existing element has been deleted from the database,the physical deletion of the existing element from memory 104 may bedelayed so that existing search threads that have a result, acquiredjust before the existing element was deleted from the database, maycontinue to use the previous state of the data. The update thread mayphysically delete (990) the existing element after all the searchthreads that began before the existing element was modified, or deleted,have finished.

Potential complications may arise from the interaction of methodsassociated with embodiments of the present invention and variousarchitectural characteristics of system 100. For example, the processoron which the update thread is executing (e.g., processor 102-1, 102-2,etc.) may include hardware to support out-of-order instructionexecution. In another example, system 100 may include an optimizingcompiler which may produce a sequence of instructions, associated withembodiments of the present invention, that have been optimallyrearranged to exploit the parallelism of the processor's internalarchitecture (e.g., processor 102-1, 102-2, etc.). Many othercomplications may readily be admitted by one skilled in the art. Datahazards arising from out-of-order instruction execution may beeliminated, for example, by creating dependencies between the creation(970) of the new element and the pointer write (980) to the database.

In one embodiment, these dependencies may be established by insertingadditional arithmetic operations, such as, for example, an exclusive OR(XOR) instruction, into the sequence of instructions executed byprocessor 102-1 to force the execution of the instructions associatedwith the creation (970) of the new element to issue, or complete, beforethe execution of the pointer write (980) to the database. For example,the contents of the location in memory 104 corresponding to the newelement may be XOR'ed with the contents of the location in memory 104corresponding to the pointer to the new element. Subsequently, theaddress of the new element may be written (980) to memory 104 to committhe new element to the database. Numerous methods to overcome thesecomplications may be readily discernable to one skilled in the art.

Several embodiments of the present invention are specificallyillustrated and described herein. However, it will be appreciated thatmodifications and variations of the present invention are covered by theabove teachings and within the purview of the appended claims withoutdeparting from the spirit and intended scope of the invention.

1. A multi-threaded network database system, comprising: at least oneprocessor coupled to a network; and a memory coupled to the processor,the memory including a database and instructions adapted to be executedby the processor to: create an update thread and a plurality of searchthreads; assign each of a plurality of search queries, received over thenetwork, to one of the plurality of search threads; for each searchthread: search the database according to the assigned search queries,create a plurality of search replies corresponding to the assignedsearch queries, and send the plurality of search replies over thenetwork; and for the update thread: create a plurality of new elementsaccording to new information received over the network, set a dirty bitwithin each of the plurality of new elements, without restricting accessto the database for the plurality of search threads, write a pointer toeach of the plurality of new elements to the database using a singleuninterruptible operation for each pointer, clear the dirty bit withineach of the plurality of new elements, and delete an existing elementfrom the memory after the pointers are written to the database.
 2. Thesystem of claim 1, wherein the instructions further include: for theupdate thread: set a dirty bit within at least one existing element tobe deleted from the database, and without restricting access to thedatabase for the plurality of search threads, de-reference the existingelement to be deleted using a single uninterruptible operation.
 3. Thesystem of claim 1, wherein the instructions further include: for theupdate thread: set a dirty bit within at least one existing element tobe modified in the database before the pointer is written tocorresponding new element, and clear the dirty bit within the existingelement after the pointer is written to the corresponding new element.4. The system of claim 1, wherein the single uninterruptible operationis a store instruction.
 5. The system of claim 4, wherein the storeinstruction writes four bytes to a memory address located on a four byteboundary.
 6. The system of claim 4, wherein the store instruction writeseight bytes to a memory address located on an eight byte boundary. 7.The system of claim 4, wherein the processor has a word size of at leastn-bytes, the memory has a width of at least n-bytes and the storeinstruction writes n-bytes to a memory address located on an n-byteboundary.
 8. The system of claim 1, wherein the plurality of searchqueries are received within a single network packet.
 9. The system ofclaim 1, wherein the plurality of search replies are sent within asingle network packet.
 10. The system of claim 1, wherein saidrestricting access includes database locking.
 11. The system of claim 1,wherein said restricting access includes spin locking.
 12. The system ofclaim 11, wherein said spin locking includes the use of at least onesemaphore.
 13. The system of claim 12, wherein said semaphore is a mutexsemaphore.
 14. The system of claim 1, further comprising a plurality ofprocessors and a symmetric multi-processing operating system.
 15. Thesystem of claim 14, wherein the plurality of search threads perform atleast 100,000 searches per second.
 16. The system of claim 15, whereinthe update thread performs at least 10,000 updates per second.
 17. Thesystem of claim 16, wherein the update thread performs between 50,000and 130,000 updates per second.
 18. The system of claim 1, wherein thepointer to the new element is written to a search index.
 19. The systemof claim 18, wherein the search index is a ternary search tree.
 20. Thesystem of claim 1, wherein the pointer to the new element is written toa data record within the database.
 21. A method for searching andconcurrently updating a database, comprising: creating an update threadand a plurality of search threads; assigning each of a plurality ofsearch queries, received over the network, to one of the plurality ofsearch threads; for each search thread: searching the database accordingto the assigned search queries, creating a plurality of search repliescorresponding to the assigned search queries, and sending the pluralityof search replies over the network; and for the update thread: creatinga plurality of new elements according to new information received overthe network, setting a dirty bit within each of the plurality of newelements, without restricting access to the database for the pluralityof search threads, writing a pointer to each of the plurality of newelements to the database using a single uninterruptible operation foreach pointer, clearing the dirty bit within each of the plurality of newelements, and deleting an existing element from the memory after thepointers are written to the database.
 22. The method of claim 21,wherein the instructions further include: for the update thread: settinga dirty bit within at least one existing element to be deleted from thedatabase, and without restricting access to the database for theplurality of search threads, de-referencing the existing element to bedeleted using a single uninterruptible operation.
 23. The method ofclaim 21, further comprising: for the update thread: setting a dirty bitwithin at least one existing element to be modified in the databasebefore the pointer is written to corresponding new element, and clearingthe dirty bit within the existing element after the pointer is writtento the corresponding new element.
 24. The method of claim 21, whereinthe single uninterruptible operation is a store instruction.
 25. Themethod of claim 23, wherein the store instruction writes four bytes to amemory address located on a four byte boundary.
 26. The method of claim23, wherein the store instruction writes eight bytes to a memory addresslocated on an eight byte boundary.
 27. The method of claim 21, whereinthe plurality of search queries are received within a single networkpacket.
 28. The method of claim 21, wherein the plurality of searchreplies are sent within a single network packet.
 29. The method of claim21, wherein said restricting access includes database locking.
 30. Themethod of claim 21, wherein said restricting access includes spinlocking.
 31. The method of claim 30, wherein said spin locking includesthe use of at least one semaphore.
 32. The method of claim 31, whereinsaid semaphore is a mutex semaphore.
 33. The method of claim 21, whereinthe plurality of search threads perform at least 100,000 searches persecond.
 34. The method of claim 21, wherein the update thread performsat least 10,000 updates per second.
 35. The method of claim 34, whereinthe update thread performs between 50,000 and 130,000 updates persecond.
 36. The method of claim 21, wherein the pointer to the newelement is written to a search index.
 37. The method of claim 21,wherein the pointer to the new element is written to a data recordwithin the database.
 38. A computer readable medium includinginstructions adapted to be executed by at least one processor toimplement a method for searching and concurrently updating a database,the method comprising: creating an update thread and a plurality ofsearch threads; assigning each of a plurality of search queries,received over the network, to one of the plurality of search threads;for each search thread: searching a database according to the assignedsearch queries, creating a plurality of search replies corresponding tothe assigned search queries, and sending the plurality of search repliesover the network; and for the update thread: creating a plurality of newelements according to new information received over the network, settinga dirty bit within each of the plurality of new elements, withoutrestricting access to the database for the plurality of search threads,writing a pointer to each of the plurality of new elements to thedatabase using a single uninterruptible operation for each pointer, andclearing the dirty bit within each of the plurality of new elements, anddeleting an existing element from the memory after the pointers arewritten to the database.
 39. The computer readable medium of claim 38,wherein the method further includes: for the update thread: setting adirty bit within at least one element to be deleted from the database,and without restricting access to the database for the plurality ofsearch threads, de-referencing the element to be deleted using a singleuninterruptible operation.
 40. The computer readable medium of claim 38,wherein the method further includes: for the update thread: setting adirty bit within at least one existing element to be modified in thedatabase before the pointer is written to corresponding new element, andclearing the dirty bit within the existing element after the pointer iswritten to the corresponding new element.
 41. The computer readablemedium of claim 38, wherein the single uninterruptible operation is astore instruction.
 42. The computer readable medium of claim 41, whereinthe store instruction writes four bytes to a memory address located on afour byte boundary.
 43. The computer readable medium of claim 41,wherein the store instruction writes eight bytes to a memory addresslocated on an eight byte boundary.
 44. The computer readable medium ofclaim 38, wherein the plurality of search queries are received within asingle network packet.
 45. The computer readable medium of claim 38,wherein the plurality of search replies are sent within a single networkpacket.
 46. The computer readable medium of claim 38, wherein saidrestricting access includes database locking.
 47. The computer readablemedium of claim 38, wherein said restricting access includes spinlocking.
 48. The computer readable medium of claim 47, wherein said spinlocking includes the use of at least one semaphore.
 49. The computerreadable medium of claim 48, wherein said semaphore is a mutexsemaphore.
 50. The computer readable medium of claim 38, wherein thepointer to the new element is written to a search index.
 51. Thecomputer readable medium of claim 38, wherein the pointer to the newelement is written to a data record within the database.
 52. The systemof claim 8, wherein the new information is received within the singlenetwork packet.
 53. The method of claim 27, wherein the new informationis received within the single network packet.
 54. The computer readablemedium of claim 44, wherein the new information is received within thesingle network packet